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ABSTRACT 

A study was conducted to observe the varying degree 
of consistency between three sources of data with respect to the same 
classroom phenomenon. Data was gathered in a public high school near 
Detroit in 1969. Using the basic 10-category Flanders interaction 
analysis system, systematic observations of controversial, issues 
discussions were made in the social studies classrooms of 14 teachers 
during 28 different discussion sessions (averaging 70 minutes per 
teacher). Student talk/teacher talk ratios (as indices of the extent 
to which students rather than teachers talk during the discussions) 
and student-initiated/teacher-initiated talk ratios (as indices of 
the tendency of students to offer their own ideas without being 
induced to do so by the teacher) were derived from the data. The 
teachers were questioned by means of a paper and pencil questionnaire 
about students* freedom to express their opinions during these 
discussions, and a sample of their students responded to a similar 
questionnaire. An analysis of the consistency of these sources of 
data was then made, using product moment correlations as indicators 
of between-source agreement. Apparent distortions in the 
teacher-reported data\seem to be related to the importance which 
teachers assign to student opinion expression as a teaching goal. The 
teacher data were found to disagree with the other two sources, and 
were judged tc be an unsatisfactory source of data for this classroom 
phenomenon. (Author/JES) 
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ABSTRACT 



This paper reports the varying degree of consistency 
between three sources of data with respect to the same class- 
room phenomenon* It is based on data gathered as part of a 
field study in a public high school near Detroit, F^.ichigan, 
in the winter semester of 1969. Using the basic ten- category 
Flanders* interaction analysis system, systematic observe-^ 
tions of controversial issues discussions were made in social 
studies classrooms* The teachers wore questioned by means of 
a paper and pencil questionnaire about students* freedom to 
express their opinions during these discussions, and a sample 
of their students responded to a similar questionnaire. An 
analysis of the consistency of these sources of data was then 
made, using product-moment correlations as indicators of bet- 
v^een- source agreement* Apparent distortions in the teacher- 
reported data' seem to be related to the importance which 
teachers assign to student opinion expression. The teacher 
data were- found to disagree with the other two sources, and were . 
judged to be an unsatisfactory source of data for this class- 
room phenomenon. 



Introduction 

A field researcher who is interested in assessing classroom behavior 
and its relationship to other variables is alv/ays faced with the problem 
of deciding from w?iat source to collect his data. At least three alter- 
natives are usually available; 1) the report of one or more outside ob- 
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servers; 2) the teacher* s report; and 3) the students* reports. For 
most pux'poses the first mentioned source is most desirable because of 
greater objectivity. The researcher is often forced because of economic 
or accessability problems to choose betv;een the latter two sources, however 
and then the question arises; Which of these is the more ‘objective?** 
Objectivity here can imply two common properties of a measure — 
those of validity and reliability. When outside observers are used in 
classrooms the reliability of their data can be checked in a number of ways 
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But teachers* reports of behavior in their own classrooms cannot be 
subjected to tests of reliability and mu.st be treated v/ith caution. 

Because in the case of student reports there can be more than one 
observation, that source might be viewed v/ith sornev/hat less concern 
on this point, although some might a*'gue that students as individuals 
are Jess able observers than teachers. How much confidence can be 
placed on teacher- and/or student-reported measures of classroom be- 
havior? This is the question wo seek to answer tentatively. 

Method and Sources of Data 

The three sources of data have as their object one social studies 
classroom phenomenon existing during teacher-identified discussions of 
controversial social issues. This is the extent to which students feel 
free to express their .opinions. That phenomenon can be referred to as 
one component of the *'openness*’of tl\e discussion, which is in turn a part 
of the ‘'classroom climate*' generated in part by the teacher, students, 
and subject matter. One observer coded class discussions live, using 
Flanders* basic ten- category interaction analysis system. Each of fourteen 
teachers in tiie social studies department of the high school under study 
was observed; twenty-eight different discussion sessions v/ere used. A 
total of 19,lb9 tallies were recorded, amounting to about 953 total 
minutes, or an average of about 34 minutes per session, and an average of 
nearly 70 minutes per teacher. Reliability of these data was checked in 
two ways. First, two taped sessions were coded by another trained observer, 
and Scott* s inter-coder reliability coefficients for the comparison are 
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0.92 and 0.85. Second, six v/eeks following the original live observations 
of the two discussions, the observer’s recoding of the tapes was compared 
with the original coding, and the reliability- coefficients are 0.74 and 
0.71 for these comparisons. 

Tv;o variables were built from the interaction analysis coding, after' 
summing all data for each teacher: (l) the sjtu.deni;t 

ratio (3 *!- 9/1 -i- 2 -i- 3 + 4 *!■ 5 t 6 -f- 7) is derived from the total number 
of 3 and 9 category tallies divided by the total tallies in categories 
1 through 7. Values of this ratio ranged from 4.29 to tOo, WD.th a rned3.an of 
.42. This yields a variable which indicates the extent to which students 
rather than teachers talk during the discussions. (2) Ihe 
tj-a ted/ stude nt talk rati^ (9/o) is derived from the 

total number of category 9 tallies divided by the total number of S’s. This 
value ranged from 34*65 to 0*46, v/ith a median of 2*43* Ihis provides a 
variable which indicates the tendency of students to offer their ovm ideas 
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v/itl) 0 ut being induced to do so by the teacher. Compared with the student 
talk-teacher talk ratio, this is considered to be a more direct indication 
of how much students actually expressed their ov/n opinions in the discus- 
sions. It is interesting to note, however, that the two measures have a 
correlation of *i-0.89, indicating a strong positive relationship. 

Questionnaire data from the teachers provided another source of data. 
Teachers were asked the following question: *'In general, curing discussions 

of current social problems, how free do you think tne class members as a 
v/hole feel to express their opinions?"; they responded by making a choice 
between: "Feel very hesitant; ""Feel somev/nat hesitant;" and "Feel free." 
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Their responses v/ere coded Ij 2, or 3 for those categories. 

Similarly, a sample of students v/ho previously had or who presently 
had one or more of these teachers was asked the same question. Tne re- 
sponses, coded as described for the teachers, were averaged to provide 
a mean student-reported score for each teacher. Numbers of students 
reporting on a given teacher ranged from 38 to *o, with a raedian of 19. 

The values for this variable ranged from 3.00 to 2.30, with a median of 
2.60. 

Two objections to the above operations must be recognized. First, 
pooling student responses to the above question into arithmetic means 
to generate a teacher’s score is not a strictly legitimate operation, 
since the level of original measurement is ordinal rather tlian interval 
or I'atio. • Because tests of significance will not be applied to the 
agreement measures (product-moment correlations) used below tnis is not 
considered a serious weakness. Second, the teacher-reported variable is 
also at the ordinal level of measurement, but this limitation is also not 
considered prohibitive since significance tests are not applied. 

One final limitation — concerning the extent of generalizability 
of findings — must be made explicit before turning to the data. This 
study is based on a limited number of teachers from a single school, and 
represents only one subject area — secondary school social studies ins- 
truction. Generalizations from the findings cannot be broad, therefore, 
although they can suggest cautions which should be heeded by researchers 
interested in studying classroom phenomena in general. 
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Findinns 

In order to evaluates the between- source consistency of observations j 
product-rnornent correlations between teachers* * and students* responses to 
the *' freedom to express opinions'* question and the student talk/toachor 
talk ratio are presented in Figure 1. The interaction analysis data ratio 
gives some indication of the relative amount of time students were allowed 



Figure 1. — Agreement between observer* s I/a student talk/teacher talk 
ratio, teacher-reported data, and student reported data on 
freedom of students to express opinions in class-’ 
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* Coefficients in this and subsequent figures are product-moment correlations. 



'-itotalk in the controversial issues discussions, and suggests an indicator 
of the amount of student opinion expression. There is agreement between 
the student talk/teacher talk ratio and the students* reports of freedom 
of opinion expression (r ~ *i-0,54), but there is disagreement betv;een the 
teachers* reports and the other two sources of data. In the case of the 
teacher data-interaction analysis data comparison, the correlation is quite 
strongly negative (r -0,80), and for the teacher- student comparison it is 
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w© 3 k).y nG 93 'tivG (r ~ •■0«38)« It is clifficult to 3 void inforiin^ to3t who 
teochers perceSvs this particular element of classro-jin phenomena diffe- 
rently than both their students and an outside observer. 

Another variable derived from the interaction analysis data is 
compared with the teacher and student data in Fi 9 ure 2. This variable 
is the student-initiated teacher-initiated student talk ratio. It is used 



Figure 2. — Agreement between observer* s I/A student- initiated/teacher 
initiated student talk ratio, teacher-reperted data, and 
student-reported data on freedom of students to express 

opinions in class 
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as a more direct measure of student opinion expression? student initiated 
rather than teacher-initiated student verbalizations are assumed to be 
more indicative of spontaneous, or "free” expression. The pattern of the 
correlations is the same as it was with the previous comparisons, with a 
strong negative correlation oetv/een teacher data and the interaccion ana— 
lysis data (r = -0.63), the same weak negative one between teacher and stuoent 
data (r = -0.38), and a positive coefficient betv-een student data and the 
interaction analysis ratio (r = +0.30). This finding strengthens the inference 
that teachers osreeive the freedom of student opinion expression dimension 
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differently than do students and outside observers* 

Explaining this finding is a rather difficult task because these 
data are taken from a study whose objectives did not include the find- 
ingj.^ let alone its explanation* One way of looking at the finding is 
in terms of teachers* assignment of irnpox'tance to the classroom goal 
of student opinion expression* If the teachex' assigns considex'aole 
importance to this part of classroora discourse, he may be more critical 
of his success in promoting such expression, and compared with his peer 
who assigns less importance, he may tend to underestimate student ex- 
pression* If this is ti'ue, it could help to explain the rather curious 



finding repox'ted above* 

Fortunately, the teachox' cjuestiorinaix'e contained another item v/nich 
is germane to this question* It asked teachers to respond to the follow- 
ing: “Students should be encouraged to voice their opinions on all 

subjects," by choosing one of the following: “Strongly Agree;" “Agree; 

“Disagree;" or “Strongly Disagree;" (these responses woK^ assigned values 
of 4, 3, 2, and 1, respectively.) Because this variable indicated the 
relative assignment of importance to student opinion expression, we can 
proceed to examine the suggested explanation given for the finding* The 
correlation betv/een the teachers* perceptions of how free students feel 
to voice opinions in their classes and the assignment of emphasis on this 
as a goal is -0*29. This tends to confirm the notion that importance as- 
signed is inversely I'elatcd to the estimation of success* It remains to 
be shov/n that a measure of- the teachers' goals agrees more closely v/ith 
the students* and observer* s data than the teacher assessment of his actual 
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success in the matter# 



Figure 3 shows that this is the case; the correlations are all 
positive, showing agreement between all three sources o£ data# We 



Figure 3# -- Agreement botv/een observer’s I/A ratios, teacher-reported 
goals for and student-reported data on freedom of students 
to express opinions in class 
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might conclude, then, that the teacher-reported data concerning the actual 
freedom of student expression in classrooms is disto3'ted by the assignment 
of importance to that condition as a teaching goal* The teacher data, 
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thorofore, should not be considered valid< 



Conclusion 

If a researcher is to select the teacher-reported data for use in 
reprecentinf: the "freedom of students to express opinions'* variable, it 
is evident from these findings that when relating it to other variables, 
he probably v/ill bo led to wrong conclusions. In the case of the pre- 
sent data, either student-reported data or data from an outside obser- 

\ 

ver is preferable to the teacher data. Researchers contemplating this 
selection problem, therefore, are advised to check the consistency of 
any teachor-generated data about classrooms v/ith external sources. 



